As known in the art, software code comprises instructions that may be used to control or instruct the operation of one or more processing devices, such as microprocessors, microcontrollers, co-processors, etc. A software development project involves several types of tasks, such as creating code (also known as coding), writing tests for the code, performing the tests, analyzing the results, and debugging. A typical software system is organized into individual units of code also known as components or modules. A software development project is organized into multiple phases, including but not limited to software design, in which specifications for the software and its constituent components are developed; software implementation, in which the code for the components is created; component testing (or unit testing), in which the components of the software system are individually tested for operability and compliance with component requirements; integration, in which the components are assembled to form the desired software system; system qualification, in which the assembled system is tested for operability and compliance with system requirements; and acceptance, in which the end user or a representative of the end user tests the newly developed system in order to determine whether to accept or reject the system.
Thus, a defect in software code may be identified as a result of testing such as Component Test (CT), System Test (ST), and a User Acceptance test (UAT). It is preferable to identify and correct a defect as soon as possible in a software development project. Defects may be introduced in any stage of a computer system development effort. For example, they may be introduced as early as the requirements stage. Such defects may be detected through reviews and tools that hold review documents such as requirements, design etc. Defect slippage, in which defects that occur in one stage occur in subsequent stages, has several impacts. Defects that occur but are not caught during CT could slip into the production version of the code and cause the system to be not accepted. In addition, the cost or effort of rework in subsequent stages may be increased. Defects may be harder to identify when code from other components is present, and a change to correct a component's defective code may not only impact code in the component but in other components as well. Further, the effectiveness of ST/UAT is reduced since testers are now bogged down with identifying CT defects.
Testing computer code is not a process of uniform complexity. Code of increased complexity is harder to write, harder to debug, harder to integrate into a component, and harder to test. In addition, components with more complex code are harder to write, harder to debug, and harder to integrate into a system of other components. In addition, the complexity of a component is a function of attributes such as the size of the component (for example, in number of lines of code) and the specified features or functionality of the component.
One measure of extent of software code testing is termed “code coverage,” which measures the degree to which the source code has been tested through use of a test suite. It is a form of testing that inspects the code directly. Code coverage measures how well the program is exercised by the test suite and thus the quality of the testing. A code coverage analysis tool may track and count the number of lines of code in the program that a test suite executed, or it may track and count “touched” features of the codes such as functions, subroutines, nodes, branch, control structures (such as IF statements), conditions and decisions. For Boolean sub-expressions, it may track and count whether the expression was evaluated both to true and false. Code coverage is defined by the developers and testers according to the needs of the project. For example, code coverage may be defined according to one or more of the tracked and counted code features described above.
State of practice defines thresholds for a development team. The thresholds may define time or output requirements of a team, or they may define code quality requirements for the project. They may also define code coverage thresholds. For example, the extent of testing conducted on a unit of code testing may be deemed acceptable for purposes of the development project if the code coverage, as it is defined by the team, reaches a selected value or values. In addition, since components are of varying complexity, time or output thresholds, code quality thresholds, and code coverage thresholds may vary with the complexity of the components of a project
Software is sometimes developed using the so-called “Agile” software development methodologies, which feature disciplined project management, teamwork, engineering best practices designed to rapidly deliver high-quality software, and a focus on aligning customer needs and company goals. Another approach is the so-called “Waterfall” methodology, in which requirements, analysis, design, coding, and testing are conducted in a highly structured, strict, and pre-planned sequence. Typically component testing in Agile projects is performed using practices such as Test Driven Development, in which a developer writes a test that defines a desired improvement or new function, produces code to pass that test, and then refactors the new code to acceptable standards. Component testing in non-Agile projects is performed typically after the code is developed.
In addition, there may be more than one approach to component testing. One approach involves testing one component at a time and completing its testing before moving on to test the next component. This approach may be called depth-wise CT. Another approach is testing a batch of components concurrently and striving to achieve high percentages of code coverage for coverage for the selected batch. The approach may be called breadth-wise CT. Both approaches have their benefits and disadvantages. For example, a depth-wise CT strategy typically results in a high level of confidence in the quality of the code in tested components, but it may also result in other components not being tested at all. A breadth-wise CT strategy typically results in a high level of confidence in the code in multiple components being essentially the same quality, but it may also result in none of the components being tested to an acceptable level.
The approach selected for use by a development team may depend on several factors. For example, if all components are equally important to the operation of the software system under development, the team may select a breadth-wise CT strategy. If, on the other hand, some components particularly need to be defect-free, the team may select a depth-wise CT strategy. It would be desirable to analyze system data to define a CT strategy so as to optimize the code coverage in the available schedule and planned effort, give the project objectives. In addition, it would be desirable to analyze testing patterns and trends to identify problems in a CT strategy underway and to amend the CT strategy in terms of schedule or planned effort according to desired objectives.
Irrespective of the practice used, the quality of the component testing is determined primarily by the code coverage—how much of the code is covered by component tests—and secondly by number of component tests that pass.
Among the technical testing challenges that development teams face is data collection. Projects may use different technologies, e.g. Net, Java and database technologies. Each technology has its preferred component testing tools. Projects find it difficult to manually collect data from these tools. Further, in most projects, there are always a number of “legacy” components that are of no interest from a monitoring and optimization standpoint. Projects find it difficult to automatically filter data from component testing tools to focus only on the relevant data. Furthermore, projects find it challenging to collate this diverse data into a cohesive and coherent form to make sense out of it and take decisions.
Another challenge is related to finding patterns in the data. Tools typically provide a “point-in-time” data for component testing. For example, a coverage tool will provide the code coverage at the time of execution. The data for that testing even is thus static, “frozen” in time. In addition, a coverage tool that is run for a longer period, say ten days, generates multiple sets of “point-in-time” data. It can be challenging for projects to identify trends in this temporal data. Further, it is can be challenging to understand the implications of trends on schedules and effort spent so far and on schedule and effort estimates.
Typically, the state of practice provides no intuitive insight available through raw data. Multi-faceted analysis of data coming out of the component testing process is not common. For example, code coverage data does not demonstrate on its face whether the project will achieve optimal component testing at the end of the estimated schedule. Further, it does not demonstrate whether a team should test multiple components at once or focus on one component at a time.
A third challenge relates to finding correlations between component testing and other phases of the project. For example, code change may occur in several phases of a project. The changes could result from an implementation of a formal change request, for example, to fix an identified defect or fault, or from a more informal code change that was not necessarily in response to an identified defect or fault. Such changes, sometimes known as “code churn,” may be defined as lines of code added, modified or deleted to a file from one version to another. Code churn may take the form of User, Date/Time, Changeset, Portfolio Project, or Path code churns.
No matter their origin or classification, code changes in one component may have unintended impacts on the functioning of the revised component and on the other components of the system. Consequently, components need to be tested after code changes, even during phases of the project subsequent to CT. Although a team might have access to information about change requests and code churn, or it may determine the existence of a trend of reduction in a component's code coverage, the change requests and code churn data and the identified trends may not be sufficient in themselves to identify potential impact on component testing.
It would be desirable to provide structured proactive guidance to code development teams related to the potential risk that is being injected into the project if CT is not managed properly, the parameters beyond coverage that may be of interest to a team, and remediation steps that could or should be taken.